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This paper suggests that while evaluation means 
ings to different people, the focus is characteristically 
been, or is being done. It is proposed that evaluation 
ied to things not yet done; that evaluation has a 
and continuous role in planning any educational program, 
s seen as facilitating the fulfillment of the four basic 
planning by (1) identifying program goals, (2) defininq 
means of attaining these goals, (3) indicating the 
sequences of selecting each alternative, and (4) 
ta for the allocation of resources. Methods of 
zing the preceding concepts, and procedures for 
priorities among goals and means are considered. The 
this data into educational planning is discusse d. (PR) 
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Evaluation means different things to different people. For example, 
it is judging the worth of a program (Scriven, 1967). It is describing 
a program as fully as possible (Stake, 196?). It is documenting how well 
program objectives are met (Tyler, 1942). It is providing useful 
information to decision-makers (Stuf flebeam, 1969). It is all of these, 
and combinations of these, and more things besides. The characteristic 
common to each of these perceptions of evaluation is a focus on what has 
been done or what is presently being done. 

We propose that evaluation also be applied to things not yet done. 
The perceptions of evaluation suggested above are not inappropriate to 
our proposal. But, in this paper we will argue, as Moynihan (1970, p. 

14) did, for "evaluation in advance." To do so, we feel, requires a 
more- than- usual reliance upon judgmental data. 
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Why Plan ? 

We’d like to talk about the planning of education programs — both 
for short- and long-range futures. Programs in education range from 
the very small (e.g., a student with a programmed text) to the very 
large (e.g., the college system of a state). A program exists over 
some period of time. Someone tries to accomplish something during that 
time. A program may be defined by the people in it, by the goals it 
seeks to accomplish, or by the reason for which it exists. A program 
is a distinguishable unit, with people, goals, and methods. 

Programs generally exist to take care of needs we have identified. 
Ideally, we identify needs that education has a responsibility to meet, 
and we devise programs that meet these needs. We might arrange Program 
A to meet Need One, Need Two, and Need Three. Program B would attend to 
Needs Four, Five, Six and Seven, and so forth until we had covered all 
needs with programs. 

Our ability to identify needs, however, is not keenly developed. 
Consequently, we nay have no program that attends to Need Two, or Need 
Five or Need Seven. We may not even know those needs exist. Further- 
more, upon examination, we discover that Program A is not satisfying 
all of Need One, or Three, or Four. There may be a multitude of reasons 



*A paper read at "Evaluation and the Planning of Educational Programs f " 
a symposium of the American Educational Research Association Meeting, 
New York, February, 1971. 
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for this failure: not enough money, or time, or experience, or personnel, 

or training. Whatever the reason. Program A does not make it. Program 
B may not fare much better. We realize that present programs may not be 
meeting present needs, either perceived or real. 

But suppose we did identify all needs. And suppose that all programs 
did satisfactorily meet all of these needs. Would this obviate the need 
for program planning? Emphatically no, since we cannot assume that 
present needs will remain constant. Need One may be met and disappear. 

Need Two undergoes alteration, resulting in a similar but not identical 
need. A series of new needs may arise, generated by changes in the 
social or educational scene, or by changes brought about by the program 
itself. The present does not remain unchanged. 

Adding further complexity to the situation is the inevitability of 
change within programs. Programs have elements of people, goals, 
methods, and time. These may not remain constant: A teacher leaves; 
spelling recitation gives way to discussion of the origin and use of 
words. Thus Program A may not be an appropriate means for meeting new 
or transformed needs. There must be a realization that present programs 
may not be responsive to future needs. 

The rationale for our emphasis on the need for planning rests upon 
three basic observations. First the speed of change of the social scene 
is a more relevant concern to educators than ever before. Many time~ 
honored traditions, ideas, and expectations are losing favor with the 
young, and are being questioned by others. The speed of change — it is 
rapidly accelerating — necessitates a different style of planning, a 
style we've not recognized before. 

Further, we're told that the amount of knowledge available to us 
is increasing at a geometric rate. There is more now known and more 
to know than ever before. Finally, because of conditions wrought by 
increased knowledge and accelerating change, man has to be much more 
psychologically and socially adaptable than ever before. If man is 
to survive, he has to be psychologically, intellectually, and socially 
fit. These observations have significance for planning education programs. 



Functions of Planning 

Educational program planning serves four basic functions. Planning 
results in the identification of potential program goals . Goals of some 
kind exist even if systematic planning is not carried out. However, 
planning seeks to sharpen goals. Goals may be stated in broad terms 
(students will become good citizens), or they may be stated specifically 
and behavioraliy (the seventh grade student will be able to identify four 
reference books). Planning enables the program manager (and his staff) 
to select particular goals, from a universe of possible goals, that seem 
appropriate for his program. "Good” planning also attends to changing 
goals as the program unfolds. 
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Planning should assist the program director in defining alternative 
means for reaching potential goals. Too often we make decisions based on 
very limited knowledge of our alternatives. If we wish to instruct 
children in the values of our society, we should consider the alternative 
means. Reading literature produced by our predecessors and contemporaries 
is an alternative. Attending to the mass media is another, but are there 
others? Can we make use of technology? Can we utilize programmed 
instruction? Should more emphasis be given to the education of the 
parents of these children? How else can we reach this goal? To be sure, 
the planning process must include feasibility checks, but wise planning 
necessitates that we have an array of alternatives before us. 

Just as planning should identify and define alternative means-to- 
ends, so should planning define some possible consequences of selecting 
each alternative. A common practice is to choose an alternative, imple- 
ment it, and then analyze the consequences of that choice. We can no 
longer enjoy the luxury of this procedure. In terms of cost alone (not 
merely dollars and cents), we need to be as fully apprised as possible 
of the consequences of our actions before we act. The plan inplies 
identifying consequences, even though this is not a simple task. We 
think evaluation could be especially relevant to this problem. 

Planning has a direct bearing on the allocation of resources . As 
alternative actions are considered, the resources required to implement 
each alternative must be considered also. What would it cost to use 
programmed instruction? What resources will be needed to accomplish a 
certain goal? New resources may be desired. Planning leads to a more 
efficient and effective allocation of resources, as systematic planning 
minimizes haphazard expenditures of funds and personnel. 



Functions of Evaluation in Planning 

Traditionally, evaluation has sought to determine the extent to 
which students in a program achieve goals set for that program. In our 
view of evaluation and planning, evaluation is broader than this. It 
should be pointed out here that the traditional use of evaluation 
remains an important component in the planning process. Description 
of goals and assessments of attempts to achieve those goals are necessary. 
Needs assessment is, in part, a measure of the discrepancy between desired 
outcomes and things as they are. To determine the discrepancy, we need 
somehow to measure things as they are. We contend, however, that evalua- 
tion has tended to emphasize too heavily a preciseness of measurement of 
present and past events, and in so doing has forfeited possibilities for 
more futuristic thought. We believe evaluation can play a wider role 
in planning. 2 

*Much of the discussion about program operation has assumed goals to be 
need based . (See, for example. Stuff lebeam’s [1969] discussion of con- 
text evaluation.) We contend that goals may also be preference based . 
That is, certain goals may be desired by various groups or individuals. 
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We have suggested four functions of program planning. Our broad 
view of evaluation suggests two basic functions of evaluation in 
planning: the examination of goals, and the examination of alternative 

means of achieving those goals, together with the potential consequences 
of those means. 

Too often, program goals are regarded as constants. Those who look 
at a program tend to consider ways the program implementation could be 
changed, not whether the goals of the program are appropriate. We con- 
tend that goals should be examined — their quality and their appropriate- 
ness ought to be assessed. This idea is not new: Scriven (1967), for 

example, made a strong argument for the examination of goals. Evaluation 
can seek to develop criteria for goal inclusion, to examine the relation- 
ships among goals, and, even more basically, to discover what people mean 
when they propose a certain goal. 

Not only should evaluation consider the quality and appropriateness 
of stated (or implicit) goals, but evaluation ought to delineate alter- 
native goals. A goal of the local school might be to get parents to the 
school. Might an alternative goal be to get the school to the parents? 
What we are striving to accomplish through our program might be considered 
in relation to possible achievements through different programs with 
different goals. What are our alternatives? Is there a pool of goals 
from which we can select? 

A discussion of goals will often eventuate in a discussion of 
priorities. Which goals are most important? Given that we cannot 
accomplish all things with our limited resources, how do we determine 
the goals to pursue? In program planning, the question of goal 
priority is critical. 

What do different groups with interests in a particular education 
program think the goals for that program should be? Do students have 
different priorities than the teachers? What of parents, and community 
members? What are their educational priorities? Do priorities change 
over time? Are our education programs sensitive to the priorities of 
those involved in the program? 

In an earlier paper (Gooler and Grotelueschen, in press), we 
encouraged, as have others, the curriculum developer to consider various 
''audiences. 11 We think curriculum developers should be held accountable 
for (to report, explain, or justify) decisions they make. The question 
of educational priorities is related to how a group or individual will 
consider program accountability. 

Any educational program tends to touch a variety of people, in a 
variety of ways, at a variety of times. These various groups can be 
viewed as audiences who come with various biases and demands, public 
and private concerns, and motives of assorted legitimacies: Each of 

these “pockets of persuasion" may serve its notice of accountability 
to the program planner or administrator. The parent has some expectations 
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as to what his child should be able to do. The taxpayer sees much money 
being spent in the school, and wonders if that money is being used most 
efficiently. Eighth graders wonder if they will be able to compete in 
high school. The amount of "clout*' these audiences possess in serving 
that notice is not at question here; the reality of their existence is. 
The manner in which the planner must account for the program will differ 
according to who is raising the questions, what those questions are, and 
when they are raised. 

We suggest that the program planner spend part of his time identify- 
ing audiences, "pockets of potential persuasion." Furthermore, it is not 
enough merely to acknowledge the existence of different audiences. Judg- 
ments made by these audiences about the goals selected for a program 
should be considered by the administrator in his decision-making. The 
administrator, or the planner, may be able to uncover (not by second 
guessing but by direct inquiry) the questions these groups will be 
asking, the claims they will be making, the axes they seem to want to 
grind. Armed with that information, the administrator, or planner, can 
determine what kinds of data should be ready for use in response to 
potential questions. Perhaps more important, the data can be used as 
developmental input; it can exert an influence on what is developed. 
Program directors cannot follow suggestions made by all audiences, nor 
should they. Suggestions may be in direct opposition to each other. 
However, by considering possible suggestions and possible demands, the 
administrator can determine, before the fact, what the consequences of 
not attending to a particular suggestion or group might be. He need not 
yield only to those who speak the loudest; he may not follow the demands 
of audiences. He does need to know what people want and think, if only 
to know better how he is ot report, explain, or justify what he intends 
to do and how he intends to do it. 

Evaluation by identifying questions, groups, and data available, 
enables the administrator to confront reality nose-to-nose . In a demo- 
cratic, pluralistic society such as ours, we cannot allow a few know- 
ledgable professionals to decide without serious consulation with other 
publics, issues of importance to the school and society. The question 
is 2 how can we best provide an educational commodity responsive to all 
who will be affected by it, and how can the educator explain that 
commodity in a manner that is understandable to the consumer? 

In addition to examining goals, we would argue that evaluation 
should seek to identify alternative means to accomplish the goals a 
program seeks. This is not an easy task. The identification and 
structuring of alternative means to ends have been hindered by our 
preoccupation with attempts to get precise measurement of what has 
already happened. Such attempts require large commitments of time 
and resources, and consequently, little effort is expended on consider- 
ing alternative ways to doing things. 
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The role of evaluation in planning might well be to gather approxi- 
mations of what has previously happened, and to identify alternative 
means to ends, based on those approximations. Planning does not and 
cannot wait until all evidence is in. Plans for next year’s programs 
are laid well before this year has expired. Traditional evaluation or 
research data has come in too late and has been too narrowly conceived 
to be of much use to the planner. Evaluators must learn how to 
approximate conditions and outcomes. Evaluation, we think, should make 
projections. From those projections will grow plans for the future, but 
these projections do not necessarily need to reflect the images of 
the past. Little of the past may be generalizable. 

When we offer alternative means to given ends, we are obliged to 
project possible consequences of selecting those alternatives: ;, lf 

you utilize these means, you might reasonably expect these consequences. M 
For example, if we use Program X in the 12th grade, your children will 
probably be more interested in vocational placement, but will emerge 
caring less about the fine arts. In addition, you may need to expect 
to retrain many of your teachers. Our present evaluation technology can 
be useful in projecting possible consequences. We’ve made some efforts 
to examine outcomes when certain activities occur in a particular 
setting. A great deal more thinking and projecting consequences is 
necessary, however. 

Evaluation has not always been viewed so broadly. The history of 
evaluation reveals a growing plea by many for the legitimation of a 
variety of data, methodologies, and responsibilities under the name 
evaluation. The development of evaluation as an independent entity is 
marked by innumerable subtle differences among those who regard them- 
selves as evaluators, with respect to what evaluation is or ought to do. 
Those subtleties are probably of interest only to evaluators. The 
not-so-subtle differences among evaluation schools are of potential 
interest to those who must cope with evaluation as a part of planning. 

Evaluation is continuous and informal. Buying a car, selecting a 
school, determining the relative merits and trade-offs of attending the 
parent-teacher meeting represent evaluative efforts. So does the 
decision to manufactur Edsels. And to stop. Evaluation is monitoring 
your existence in your society. 

Evaluation has been viewed as an appendage to the operation of a 
program, as opposed to an Integral part of it. We would argue that eval- 
uation must be integrated into program planning and operation. Evalua- 
tion must begin early; it must monitor it must project. 

So what have we said about evaluation and planning? We’ve indicated 
that planning, always an important component in the operation of a program, 
may need to assume new methodologies if it is to function efficiently in 
a rapidly changing society. We’ve said we’re uncomfortable with our 
present ability to identify needs, or to attend to all those needs. The 
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specification of alternative goals, or means, or consequences , is an 
important aspect of planning. No program can do all things; priorities 
must be determined. Adequate planning, we think, leads to more efficient 
and effective allocation of resources. 

Evaluation has traditionally focused on the past. Evaluation has 
meant standardized tests and grades and accountability. We think it still 
means this, but it also means much more. Evaluation plays a role in 
planning by examining goals and means — future goals and means. We've 
argued that evaluation can approximate conditions and outcomes of the past, 
and, based in part on those approximations, can project to the future. We 
would opt for less effort expended to obtain precise measurement, and more 
effort to structuring alternatives for the future. We recognize a trade- 
off: mistakes may occur if we don't knovr why something done previously 

worked or didn't work. We are in danger of repeating mistakes if we only 
approximate the past in attempts to study the future. Wardrop (1971) 
suggests that an important concern of educational evaluation is explanation 
— the determination of the most probable cause for a phenomenon. We 
contend, however, that explanation is extremely time consuming, and that 
decisions about programs may need to be made before explanation is complete. 

The apparent broadness with which we view the function of evaluation 
in planning may cause some of our colleagues to feel we've been talking 
about something other than evaluation. Is the structuring of alternative 
goals and means and consequences a measurement of student outcomes, or 
an assessment of the worth of something, or the collection of useful 
information for the decision-maker? Perhaps not. Perhaps evaluation 
is not the most appropriate term for the procedures and purposes we are 
proposing. We will search for a more appropriate term. 

We need to look at programs with a fish-eye lens. By so doing, we 
can see what's behind us, but then we also have a wide view of what may 
be before us. 

In the next section of this paper, we'd like to talk about how we 
might operationalize some of these ideas. 

We will stress the importance of ascertaining alternative goals and 
means. Further, we will suggest procedures to determine priorities among 
potential goals and means. Finally, the inclusion of these data into 
educational program planning will be discussed. 



The Importance of Alternatives 
It is a tale , 

Told by an idiot 3 full of sound and fury , 

Signifying nothing . 

Macbeth 3 Act Vj Scene VI 

There are many such tales. And most of us fear that our ideas may 
be so described. Those involved with the planning of education programs 
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ie us t be prepared to risk this Skakespearean evaluation of their ideas, 
however, they must also be willing to examine alternatives to what is 
presently being done . 

Buy why? Aren't the people with a vested interest in an education 
program (teachers, parents, kids) in agreement with the goals and means 
of a program? Most likely not, particularly if that program is yet to be 
designed. People have different viewpoints about what a new program 
might accomplish. They also have different ideas about means to be used 
to accomplish a program's goals. 

One role of evaluation in the planning of a program is to describe 
the goals constituents think should be pursued in a program, as well as 
the variety of goals seen as desirable by people not directly involved 
in a particular program. Evaluation is not obliged to derive consensus. 
Divergence may be highly desirable: we think it is essential. 

It is easy to view our society and our programs from a narrow 
perspective, and it is possible to remain comfortable with that view if 
v/e are never forced to consider alternatives to the traditionally held 
goals. We have argued that change is constantly accelerating. Our 
perception of the world as we have always viewed it may not be a valid 
perception of the world that now is. We think the individual who considers 
many goals may be more likely to examine what he is now doing in terms 
of what he could be doing, than an individual not confronted with these 
alternative goals. 

Perhaps even more important is the process of habitually scrutini- 
zing alternative goals. This involves a continual examination of goals, 
programs, and society. 

Ziegler (1969) has suggested that planning is the traditional method 
for attempting to impose some order on the future. The future may be 
similar to the past in many ways. The future is not the past, however. 
Given the present and probable future rate of change, it is more and more 
unlikely that the future will be a replication of the past. The alterna- 
tive goals we delineate must be indicative of that which jls and will be 
as opposed to that which was. We are suggesting that less emphasis in 
planning should be on the past. 



Goals and Priorities 

Goals have been held, discussed, and debated since the beginning of 
man. They exist at many levels. There are personal goals, professional 
goals, and goals held to be "for the good of society." Goals range from 
the highly general (to lead a productive life) to the highly specific 
(the student will be able to name the Presidents). We have been urging 
planners to consider a wide variety of possible goals. The array of 
possibilities should reflect the specific and the general, as well as the 
insiders* and the outsiders' views. And the goals must be stated in such 
a way that their intent is clear to the people who will judge them. 
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What should an education program do? A response to this question 
implies priorities. How do we structure an education program when it 
must operate under constraints, making impossible the pursual of some 
of those goals we recognize as good and useful? Priorities are the 
assignment of relative worth to two or more functions or ideas. 

Priorities imply choice. To have a choice, we must have a number of 
goals to choose from. 

Priorities are formulated and operationalized constantly. We do 
it everyday, mostly implicitly. Sometimes explicitly. We decide to go 
to the store, as opposed to staying home. We select a particular tele- 
vision program. We choose certain food. But we are urging the education 
planner to make more explicit the implicit. We’re asking that people 
think about priorities. In the next section, we'd like to describe 
how information about goal priorities might be collected and used. 



Profiles of Goal Priorities 

A number of techniques may be used to collect information about the 
priority various people assign to particular goals of education programs. 
Some of these techniques will be illustrated below. 

Sorting . Downey (1960) used a technique called the Q-Sort to deter- 
mine how important 16 educational goals were rated by a variety of people. 

Using this method, the investigator asks the respondent to sort a 
series of possible goals into categories according to perceived importance 
of each goal. The respondent is allowed to put only a certain number of 
goal statements into each category. If there were 16 goals, as Downey 
had, the categories would be as follows: 

Host Important 

12 3 4 



Each box represents one goal statement. Thus, the respondent must choose 
one goal as most important, two goals as next important, and so forth. 

The average ranking assigned to each goal by the group of people 
being questioned can be computed. On the basis of these rankings, the 
entire group of goal statements can be ranked for each group of people. 
The results may be plotted as a profile. Downey developed a profile 
for responses to his 16 goals, as illustrated in Figure 1. Another way 
of presenting the information would be simply to list the goals accord- 
ing to average ranking. 




Least Important 

5 6 7 
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0 1 2 3 4 5 6 7 

Knowledge 

Intellectual Skills 

Creativity 

Desire for Knowledge 



Man to Fellow Man 
Citizenship 
Partiotism 
World Citizenship 



Physical 

Emotional 

Ethical 

Aesthetic 



Vocation-Selective 
Vocat Jon-Preparative 
Home and Family 
Consumer 



Figure 1: Relative Importance of the High School Task Elements. 

(Taken from Downey, I960, p. 34.) 

The importance of divergency in goal statements is again apparent. 
To know what people regard as important, we need to know how they will 
rank a variety of goals. 

Rating . Respondents might be asked to rate a particular goal on 
a continuum of importance. Consider the following as an example of a 
format for asking about the importance of possible goals of a secondary 
school program: 



Below are some goals for a secondary 
school. How important do you think 
each of these goals should be ? 

This is ji goal the school should : 

Carefully Consider Consider Consider 

avoid relatively relatively extremely 

pursuing unimportant important important 



To provide the student 
training and placement 
in the vocation of his 
choice. 

To foster the develop- 
ment of the student as #' 

an enlightened consumer. 

♦ * Students 

To foster in the student • Parents 

emotional stability and 

good mental health. 
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A mean ratine could be calculated for each of the goals and a profile 
drawn for a particular group or individual. These ratings could also be 
plotted for various groups as shown above. 

The investigator may wish to note those goals consistently rated 
as very important, and those goals people feel should not be pursued in 
the program. Frequently he will choose to compare the profiles of 
various groups. 

Allocation of resources . It is possible that an individual, con- 
fronted with 10 or 17 or 6? goals for an education program, might rate 
all of the goals as "very important . " In the rating technique outlined 
above the respondent is not forced to rank goals, as he was in the 
Q-Sort exercise. 

Resources are finite, however. Wo program can accomplish everything. 
One way of gaining some insight into what people think is important is 
to ask them to allocate these finite resources among goals. 

One way of allocating resources is to ask people what percentage 
of the total amount of money available for a given education program 
should be allocated to pursuing each of a number of goals. The respon- 
dent might also be asked how he would allocate time to pursuing certain 
goals. Those goals receiving greater amounts of time or money resources 
would be considered more important than goals receiving fewer resources. 

Suppose an individual were given a series of goals potentially 
appropriate for the education program of a certain community. The 
individual might be asked to respond to an item such as the following: 



Resources, both time and money, are often not as plenteous as 
we would like them to be. Educators try to pursue as many 
goals as they can with the time and money available. Suppose 
the school were required to cut back drastically on its program. 
What educational goals are most important for the school, if 
they are not able to pursue all of them? Below, list the 
number of the goal that best completes each statement. 

If the community had enough resources to pursue: 

1 goal, it should pursue Goal 

2 goals, it should pursue the previous goal and . . Goal 

3 goals, it should pursue the previous goals and . , Goal 

4 goals, it should pursue the previous goals and . . Goal 



The strength of resource allocation as an indicator of priorities 
lies with the familiarity most people have with some kind of resource 
allocation. Most of us have to decide how we will spend our time and/or 
our money. We have learned some things about manipulating both kinds of 
resources. (Wote : See Gooler [1971] and Wilder [1968] for more detailed 

“~nples of priority data collection.) 
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Two points should be made concerning the brief description of 
methods we have outlined for collecting priority information. First, 
the usefulness of priority information will be largely dependent on the 
quality and quantity of the goal statements people are asked to respond 
to. If respondents do not have many alternatives to choose from, the 
investigator may not obtain as complete a picture of what people think 
is important as he might if people were required to respond to a variety 
of possible goals. 

Second, we feel that the most complete representation of what people 
think is important may include a look at priorities from several perspec- 
tives. An individual may order a set of goals one way when asked to talk 
about those goals in terms of importance ; he may order those same goals 
differently when asked to allocate resources to those goals. No one 
ordering may be more or less accurate than another, but are rather 
different . It is important to look at the differences as well as the 
commonalities . 

We have suggested one kind of relationship that may exist among 
goals: a rank ordering relationship. Other relationships may also 

exist. For example, some goals may be related in that the inclusion 
of one goal assumes the exclusion of another goal. Goals may be grouped 
together according to some common base. Thus, one group of goals may be 
consiaered more important than some other group of goals. Each of these 
relationships may be important to examine. 



Using Priority Information in Program Planning 

Our brief discussion of methods of collecting information about goal 
priorities has been, at best, suggestive of ways of collecting priority 
information. But, how is the information to be used once it has been 
collected? 

A study of priorities will reveal concerns and disagreement among 
various groups of people as to what education ought to be doing. The 
program planner can use this information in several ways. First, he may 
plan programs that will attend to those things people think are important. 
Second, where he detects disagreements he may plan ways to lessen that 
disagreement. He may need to supply additional information to the 
disagreeing groups. He also might enlist the aid of advocates to clarify 
particular program goals about which there is disagreement. 

At best, knowledge of where disagreements may occur will enable 
the planner to lay a careful basis for the implementation of a program 
so that its goals might be achieved. At worst, the planner will know 
where he might expect resistance to the implementation of a certain 
program. To be sure, the information obtained about goal priorities 
allows the planner to seek means for actively reflecting or obtaining 
program support. 





